Improve error message and add FAQ entry for stratified CV with Dataset (#1122) - #1152
Open
Sweeyya wants to merge 1 commit into
Open
Improve error message and add FAQ entry for stratified CV with Dataset (#1122)#1152Sweeyya wants to merge 1 commit into
Sweeyya wants to merge 1 commit into
Conversation
Fixes the unclear ValueError raised by ValidSplit when a skorch Dataset or TensorDataset is passed to .fit() with y=None and stratified=True. The error now names the dataset type and points to the documented workarounds (predefined_split, stratified=False, or train_split=None) instead of a generic message. Refs skorch-dev#1122
BenjaminBossan
requested changes
Aug 10, 2026
BenjaminBossan
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for this PR to improve the error message. There is one merge conflict, could you please resolve it? Moreover, there is one more option for users if they encounter this error. This would affect the docs and also the error message.
| .. code:: python | ||
|
|
||
| net = NeuralNetClassifier(MyModule, train_split=None) | ||
|
|
Collaborator
There was a problem hiding this comment.
Yet another option is to explicitly pass y, if the user can somehow retrieve it. Something like this would work: net.fit(dataset, y=y)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
skorch's docs say you can pass a PyTorch or skorch
Datasetdirectly to.fit(). Doing so currently fails with a confusing error(
ValueError: Stratified CV requires explicitly passing a suitable y),because the default validation split logic (
ValidSplit, stratified bydefault on
NeuralNetClassifier) can't pull class labels out of aDatasetobject the way it can from raw arrays. Users hitting this had no indication
of what to do next (see the StackOverflow thread linked in the issue).
Per @BenjaminBossan's guidance on the issue, this fix does not guess
yfrom the dataset or silently disable stratification. It only makes the
existing error message and docs more actionable.
What changed
ValidSplit.__call__now detects when the failingdatasetis askorch.dataset.Datasetortorch.utils.data.TensorDatasetand extendsthe error message to name the type and point to three workarounds:
predefined_split(),train_split=ValidSplit(5, stratified=False), ortrain_split=None. Any other type still gets the original message,unchanged.
docs/user/FAQ.rstcovering this failure and thesame three workarounds.
CHANGES.mdentry under Unreleased.Closes #1122
Acceptance criteria
skorch/tests/test_dataset.py:Dataset,TensorDataset,and generic array cases)
python -m pytest: 1648 passed, 104 skipped,1 xfailed)
dataset.pyand the testfiles
detected dataset types, where the message is strictly more
informative
Before and after
Before:
ValueError: Stratified CV requires explicitly passing a suitable y.
After:
ValueError: Stratified CV requires explicitly passing a suitable y. You
passed a skorch.dataset.Dataset as dataset, which skorch cannot introspect
to obtain labels for stratification. To resolve this, either wrap your
validation data with skorch.helper.predefined_split and pass it as
train_split, disable stratification with train_split=ValidSplit(5,
stratified=False), or disable the internal validation split entirely with
train_split=None.
Testing
test_net.pytest that hard coded the oldmessage text
new message
sphinx-buildto confirm the FAQ entryrenders correctly
@BenjaminBossan, ready for review whenever you have a chance. Thank you for
scoping this out and for the mentorship on my first contribution here.